2. SLS Lecture 2 : Shell Part II¶
Getting our hands dirty.
As powerful as UNIX and the shell are they are also old and dirty... have had plenty of time to accumulate some Barnacles.
Our goals:
Basic proficiencies in
using the shell as a programming language
core to automating your UNIX life
interactively : one liners
scripts : small and large programs
navigating and working with the file system
running and managing running programs
IO redirection and pipes
processes
understand basics of UNIX credentials
user and groups names and ids, file permissions, and who “root” is
Enable you to learn more
know enough so that you can experiment
know where you can look for more info
know enough so you can appreciate and learn from others
idioms, tricks, and hacks
scripts
Understand how things work so that you have the context for
working efficiently with large bodies of ascii files
at the command line
and in editors like
vimandemacs
makegit
2.1. The Shell as a Programming Language¶
** The Glue that holds the UNIX Universe together **
Basic Syntax : Notes
comments
echo and printf
simple variable syntax and usage
quoting to control splitting and expansion
many more fancy variable expansions and special variables :
help variablesbut plenty more
basic programming constructs
for lists of words
commands and exit codes
(())
[[]]
[ ]
interactive vs scripts
history and arrow keys
scripts
ascii file with lines of shell commands
mark as executable and put in path or name explicitly
3. Shell Programming¶
# Comments
# a word that starts with # and everthing else till end end of the line is ignored
# echo - basic built in for outputting
echo hello
# shell variables and variable expansion has lots of subtly but lets start simple
i=hello
# remember shell expansions will happen to the line prior the "echo" built in being
# executed
echo $i # $ -- triggers parameter/variable expansion
# quoting used to control expansion and splitting
x=goodbye and farewell
# single quotes supresses all expansion and splitting - literal value of each character
# single quotes cannot contain single quotes
x='goodbye and farewell'
echo $x
x='$i goodbye and farewell'
echo $x
# double quotes partially suppress expansions but allow some like $
x="$i goodbye and farewell"
# ; can be used to join commands as a list on the same line
x="$i goodbye and farewell"; echo $x
# ok we will look at more variable related syntax and expansions later
# lets try out some loops: for, until, while, break
# for name in word ; do commands; done
for color in red green black brown; do echo "color=$color"; done
# or
for color in read green black brown
do
echo "color=$color"
done
# the shell also has support for integers and arithmetic
for ((i=0; i<5; i++)); do echo "i=$i"; done
# actually (()) tells the shell to do arithmetic in general
(( j++ )); echo $j
# shell arithemetic supports comparisons as well
(( j<0 ))
# special parameter ? is exit code of last command/pipe
echo $?
(( j>0 )); echo $?
# shell convention is exit value of 0 is success and non-zero if failure
# becare oposite to what you might be used to
if (( j == 0 )); then echo "j is zero"; elif ((j<10)); then echo "j is under 10"; else "j is something else $j"; fi
# beyond arithmetic expressions bash support a more general [[]] expression
# return code will still indicate success vs failure bue allow non-integer logic
[[ $i == "hello" ]]; echo $?
# special case of logical and and or to conditional commbine commands
(( j<0 )) && echo "j is < 0"
(( j<0 )) || echo "j is >= 0"
# could have written using if
# conditions are just tests of exit code of command eg. success versus failure
# So any command (internal or external) written to correctly indicate success and failure with exit code
# can be used as condition
# shell also has case and select statements
4. File and Directories¶
# Current working directory
pwd
# What is in the current directory?
# Another expansion - Filename expansion the shell does for us.
echo *
echo /*
# ls is a powerfull external command for list contents of directories
ls
man ls
ls -l
ls -lrt
ls -lhrt
ls -1
# change working directory
#
cd /
pwd
ls
cd home
pwd
ls
cd jovyan
cd /home/jovyan
pwd
cd /usr
pwd
ls -l /var/*
pwd
cd
# . and ..
ls .
ls ..
# full path
ls /bin/ls
ls /var/log
# combining
cd /var/log/../g*
# short cut to name home
# ~ expansion
echo ~
ls ~
echo ~*
# . prefix name to hide
ls -a
# creating directoreies
# man mkdir
mkdir ~/bin
mkdir $HOME/tmp
cd ~
mkdir Classes
mkdir Classes/CS
mkdir Classes/CS/400
MYDIR=Classes/CS/400
for ((i=0; i<10; i++)); do echo mkdir $MYDIR/$i; done
# create an empty file
# man touch
touch foo
ls
rm foo
#for ((i=0; i<10; i++)); do echo touch $MYDIR/$i; done
for d in $(ls $MYDIR); do rm $MYDIR/$d/README.md; done
rm -rf Classes
# find is also a powerfull command
4.1. Running and Managing Programs (Commands)¶
4.1.1. Processes, Files and Streams¶
What is a process in more detail and what can it do
a running program
process can launch another processes
bash is implemented to start commands as process (when needed)
passes command line arguments and environment variables
Processes, Files and channel/stream model of I/O
files and Kernel objects
read, write
everything is a file
Streams: Processes and Files
open : attach a file as a stream
file descriptors/handles : stream
read and write bytes to a stream
close
dup
standard input, standard output, standard error
Shell Syntax:
echo 'Hello world' > hellocat hellocat < ./hello
We now can understand what a pipe is
pipe file object
Process 1 stdout into pipe and Process 2 stdin from pipe
Shell Syntax:
‘ls -1 | wc -l’
‘ls -1 | grep ‘^l*’ | wc -l’
mknod mypipe pandmkfifo mypipe
4.2. Credentials and file permissions¶
Process have id’s associated with them
a single user id :
ida single number that maps to a string user name (
/etc/passwd)
set of group ids
user has a primary group but can be in many secondary groups
each has a number that maps to a name (
/etc/group)each group can have many users
ps auxgwwprocess’s inherit their ids from their parent
Files have id’s and permissions
ls -l,chmod,chown, andchgrpuser, group, other -> read, write, execute
kernel ensures that process id’s and requested operations match permissions
Process management
ps- look at all processesThe shell and its children
&: foreground and backgroundctrl-zjobs
so we know how to start process, list them how about stopping
killsignals
ctrl-cor without prejudice
4.3. Variables in detail¶
has a name and stores a value
variables are assigned a value with
name=[value]Note cannot have spaces between either name
=or=and valueby default variable are “string” variables
NULL/empty string is a valid value eg.
x=value of an assignment is subject to the following expansions:
tilde
parameter and variable expansion
quote removal
if variable has integer attribute then arithmetic expansion is done
one more attributes that can be assigned with the
declarebuiltinno need to declare
eg. assignment to a non-exiting variable creates it
can mark some variables for export (environment variables)
“exported” variables are copied via UNIX environment to child processes
Environment a set of strings of the form : “NAME=STRING”
Bash has the kernel copy the current set of exported variables when it launches a new program. Programs can inspect these values and customize their behavior. (eg. locale)
export <Variable>[=Value](can do the same withdeclare -x)
variable expansion and quotes -
echo $x $yvsecho 'echo $x $y'vs echo “\(x \)y”-echo “$x $y”`some special unnamed variables (called parameters)
actually variables are technically parameters with names
1-n– positional parameters expand to ordered arguments of script or function@and*– expands to positional parameters however with differing expansion rules#– expands to number of positional parameters?– expands to exit status of last command/pipelineothers –
-,$,!,0
Advanced:
declareintegers
declare -i myvar; myvar=myvar+1; echo $myvarvs not using the declare -i?
indexed arrays
associative arrays
ref variables
other built-ins that are special forms of
declaretypeset(same asdeclare)export(like usingdeclare -x)readonly(like usingdeclare -r)+=unset
things that look like variables but are really fancy expansions
$actually triggers three types of expansionparameter expansion
command substitution
arithmetic expansion
Parameter expansion
simple:
$nameor more explicitly${name}many advanced forms: Here are a few that I constantly use
setting default value eg.
foo=${foo:-START}delete prefix eg.
foo=ABCA.txt; echo ${foo##*.}delete suffix eg.
foo=ABCA.txt; echo ${foo%%.txt}substitution/deletion eg.
foo=ABCA.txt; echo ${foo//A/O}orecho ${foo//A/}
Command substitution
very useful run a command and treat its output as a value
foo=$(ls); echo $lsorn=$(ls -1 | wc -l); echo $n
Arithmetic expansion:
$(()), or(())or let, or if integer types variablesShell knows how to do basic integer math and some binary/bit operators
++,--,+,-,!,~,**,*,/,%,<<,>><=,>=,<,>,==,!=&,^,|,&&,||expr ? expr : expr=,*=,/=,%=,+=,-=,<<=,>>=,&=,^=,|=number formats:
prefix:
0- octal,0x- hex,[#base]number– eg.((x=2#101)); echo $x
See. https://www.gnu.org/software/bash/manual/bash.html#Shell-Commands 2. Breakdown complex commands into simple commands
complex commands are a composition of simple commands:
pipelines
connecting inputs and outputs of commands together using
|ls -1 /bin | wc -l
lists
one or more pipelines separated by one of:
;,&&,||or&and terminated by;or;or&or a newline.echo "about to run ls"; ls; echo "done running ls"cmd1 && cmd2: runcmd2ifcmd1returns “success”cmd1 || cmd2: runcmd2ifcmd1returns “not success” aka “failure”cmd1 & cmd2: runcmd1asynchronously in the background and runcmd2“normally”lots more to say about asynchronous commands
compound commands: Shell programming constructs (these can span lines)
loops:
until,while,for,do,done,break,continuethere are two type of
forloops
conditionals:
if,then,elif,else,fi,case,select,(()),[[]]grouping:
(),{}
functions
/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 1 \(+\) Problem 1
/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 2 \(+\) Problem 1
/ \(+\) home \(+\) jonathan \(+\) Classes \(+\) CS \(+\) 210 \(+\) Assignment 3 \(+\) Problem 1
“Full” path name: join independent components with the “/” character. So the above three files as proper unix path names would be: